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ABSTRACT 

A new federal initiative calls fora "College Scorecard''which will include a yetto be determined measure of graduate earnings. In 
this paper we examine the political context that drives this initiative and examine the nascent efforts of four states to develop 
statewide systems to measure the labor market outcomes of higher education. We propose five principles to support a system 
that would generate valid labor market measures that could cut across all segments of higher education in California, and 
disaggregate down to campuses, departments and programs. We present results from a large-scale pilot project using these 
principles that generated labor market outcomes for 44,000 college students from California State University Northridge. Finally, 
we recommend an agenda for creating a statewide system to measure labor market outcomes in California. A first step is to 
create a venue in which policymakers representing the three public segments (University of California, California State University 
and the California Community Colleges) and the private sector can come together to design a system. In the past, the California 
Postsecondary Education Commission (CPEC) would have been a natural body to do this, as it had representatives from all 
higher education segments. But it was essentially closed in the last round of budget cuts, so an alternative is needed. We 
believe that a system that emerges through voluntary cooperation will be stronger in the long run than one that is imposed 
legislatively. We recommend that each sector designate a high level administrator to join an ad hoc group to develop a system 
that will ultimately be endorsed by the boards of each system. Private institutions should be represented as well and should 
encourage their colleagues to join in. 


On j anuary 27, 2012 at the University of Michigan, President Obama laid out his vision for a federal initiative that would reward 
colleges and universities with increases in federal student aid if colleges restrained cost increases and increased their 
accountability to the public.^ Atthe heart of this initiative is a new "College Scorecard" that, according to the White House, will 
"assist prospective student and their families in comparing colleges before they choose using key measures of college 
affordability and value. There are five measures in the score card, beginning with the cost of attendance, a series of four 
graduation rates, student loan repayment rates, student loan debt, and of most interest to us an "Earnings Potential" measure. 

The "Earnings Potential" measure, unlike the other measures in the scorecard, is not specified. The model scorecard simply 
says the following about the measure: ‘We don't have this information yet. Before you enroll, ask [your university] to tell you 
about how many of their graduates get jobs, what kinds of jobs they get and how much they typically earn."^ While few colleges 
and universities could produce such data now, a valid, reliable and reasonably cost efficient method for measuring earnings is 
closer than most people in higher education know. This paper reports on a pilot project which followed up 44,000 students who 
entered California State University, Northridge during the 1995-2000 period to measure their earnings one year and five years 
after leaving higher education using administrative data sets available in every state. 

In this paper we briefly outline the renewed interest in the labor market experience of college students and then describe in detail 
the method we developed for measuring the earnings and labor market experience of college students, both graduates and 


1 The authors would like to thank Provost Harry Hellenbrand, of California State University, Northridge for taking a risk and supporting the 
project on which this paper is based. We also thank our colleagues at Labor Market Information Division of the California Employment 
Development Department, Steve Saxon, Spencer Wong, Andrew Wong and ] ennifer Taylor who contributed to all aspects of ourproject. 
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dropouts. The system we developed can produce earnings data overtime down to the department and program level. Finally, 
we lay out an agenda for developing a state wide system in California that can produce labor market outcomes on an on-going 
basis, objectively and cost efficiently, for all segments of higher education, and fill in the blank in the President's scorecard. 

A. THE IMPORTANCE OF MEASURING LABOR MARKET OUTCOMES 

The President's proposed scorecard did not drop out of the blue. Going back to 1983 when the A Nation atRisk report was first 
published, all of education has been under increased scrutiny, from analysts, policy makers and the public. Initially attention 
focused on K through 12 education, but as evidence mounted that the US higher education system was falling behind other 
developed countries, the attention of critics and reformers turned towards higher education. In a 2011 article Charles Kolb, 
President of the Committee for Economic Development, makes an increasingly common critique of the higher education system 
in Change: The Magazine of Higher Learning. 

American postsecondary education has avoided the accountability spotlight. Our postsecondary policy debates have 
focused mostly on input problems such as access, the cost of the federal student loan program, the value of the Pell Grant, 
and diversity. Issues such as graduation rates, the quality of learning, and cost effectiveness were rarely addressed: 
Everyone simply assumed that America had the best postsecondary education system in the world... We have gone from 
first to ninth among developed nations in the 33-member Organization for Economic Cooperation and Development^Pg. 
14) 

In truth, higher education has not completely avoided the "accountability spotlight"; in 2005 the State Higher Education Officers 
issued a special report entitled Accountability for Better Results: A national Imperative for Higher Education calling for an 
increased focus on the performance of higher education. = A recent study found that in face increased demand for higher 
education and reduced funding many policy makers turned to performance based funding to prod higher education toward 
specific policy goals. But while research shows that half the states had tried some form of performance based funding, half of 
those who tried it abandoned it and the rules and methods used were unstable®. 

In two states with the longest running performance funding programs labor market outcomes were included in performance. In 
Florida quarterly earnings for graduates six months after graduation was one performance measure. In Tennessee placement in 
employmentwas a performance measure for community colleges.'' 

The issues raised by these critiques are particularly acute in California, where a continuing budget crisis, shifting demographics 
and a rapidly changing economy have combined to undermine affordability, access, and the capacity of the system to support 
the state's economy. As a result of this critique far reaching proposals are emerging to significantly reform the system®. Almost 
all of these proposals include changes to better align higher education with the needs of the economy. These proposals in turn 
raise the question of how will campuses and policy makers know if they are better serving the economy, if they don't know where 
their students end up in the labor market? 

As the nation continues to wrestle with slow economic growth and high unemployment, the questions of whether or not "college 
pays off" is on the mind of policymakers, parents and students. For example, a widely reported study out of the Georgetown 
University Center for Education and the Workforce, titled Hard times: College Majors, Unemployment and Earnings, Not all 
College Degrees are Created Equal used national data from 2009-10 to ask the question "Is college worth it? "A The answerwas 
yes, but, as the authors went on to point out, unemployment among recent college graduates was at record highs, and those with 
graduate degrees do not always out earn those with bachelor's degrees. These authors worked with standard national data sets 
and hence could not bring their analysis down to the campus level. 

There is a long history of research showing that college graduates do indeed earn substantially more than workers without a 
college degree'®. But critics, parents and current students have other questions, such as: Does college pay off for someone like 
me? Will I earn more or less depending on which campus or major I choose? Which college experiences affect how much I will 
earn? Will the increasing number of students in need of remediation achieve the same earning increases as other students? 

In our view, it seems inevitable that in the current climate higher education institutions will have to produce some earnings and 
labor market data for their programs, whether the President's scorecard is implemented or not. The question that concerns us is 
how will this be done? Will the federal government or accrediting bodies impose an arbitrary and possibly poorly conceived 
measure? Or will each campus initiate some sort of survey of graduates, so results and populations will vary from campus to 
campus and state to state? We recommend another path. We believe that higher education institutions and systems can come 
together and develop a meaningful, comparable measure of earnings, and other labor market outcomes that can both provide 
public accountability and provide campuses with the data they need to make critical decisions. 
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B. PRINCIPLES FOR AN EFFECTIVE LABOR MARKET OUTCOMES SYSTEM 

Based on our experience, which we eiaborate beiow, we beiieve an effective system that can cut across states and types of 
institutions shouid adhere to five principies. 

1. Foiiow aii matricuiated students overtime. 

This is a criticai feature of an effective system. The sad fact is that nationaiiy about 40% of student enroiiing in four year 
institutions don't graduate. The proportion is higher in the community coiiege sector. These students consume pubiic resources 
and invest their own time and money as weii. We need to know what happened to them after they ieft. Simiiariy some students 
transfer to other institutions and compiete there, campuses shouid know how they did in the iabor market even if they did not 
compiete at their originai institution. Without tracking graduates, transfers, and ieavers, the system wiii not be abie to show the 
benefit of graduating over not graduating. 

We know the benefits of higher education are iong term. Oniy measuring earnings and empioyment in the first or second year 
after graduation, ieads to systematicaiiy underestimating the impact of higher education. Recent graduates often flounder in the 
iabor market for sometime after graduation. Their advantages over non-graduates emerge as they find opportunities that are a fit 
for their skiiis. Simiiariy, gaps between graduates in technicai fieids and iiberai arts fieids are iargest right after graduation and 
often diminish overtime. To getthe whoie picture we have to see how earnings and empioyment evoive overtime. 

2. Use standard data sources avaiiabie in every state. 

Comparabie measures cannot be caicuiated uniess the same data are used in each state. We have identified standard data 
sources tied to the federai unempioyment insurance system and other sources that can be used across the nation. We wiii 
eiaborate them iater in the paper. 

3. Create standard, easiiy understood iabor market measures. 

The President's proposai essentiaiiy caiis for consumer information to inform students, parents and other interested parties about 
the economic outcomes of higher education. This means the measure must be quickiy and easiiy understood by non-speciaiists. 
Hence measures shouid be few and expressed in terms thatpeopie can understand without eiaborate expianations. This means 
measures that may be different from those used in the schoiariy research on this subject. 

4. Disaggregate data to the campus and program ievei. 

We iargeiy know what the typicai benefits of higher education are. What we don't know are the outcomes for specific campuses 
or programs. To meet the demand for detaiied information that can inform individuai decisions and be used to improve 
campuses and programs, data must be disaggregated down to these ieveis. This means that we need a system that can 
essentiaiiy foiiow aii students on a reguiar basis, and report resuits quickiy and efficientiy to academic units and other 
stakehoiders. At the same time, institutionai stakehoiders in particuiar must be sensitive to the difficuity of creating accurate 
estimates for smaii departments and degree programs. 

5. Make resuits pubiicaiiy avaiiabie. 

A key point of the President's score card is that the data wouid be readiiy avaiiabie for aii interested parties. This means that 
resuits aiiowing comparisons between systems, campuses and programs must be readiiy accessibie to typicai users. In the 
foiiowing section we take a iook at initiai efforts by four states to generate measures of empioyment and earnings and evaiuate 
them againstthese five principies. 

C. WHAT ARE LEADING STATES DOING? 

Using personai contacts, we identified four states that have iaunched at ieast piiot efforts to coiiect anaiyze and report earning 
and empioyment data for higher education. They inciude: Fiorida, Minnesota, Washington, and Virginia. We acknowiedge that 
there may be other states thatwe have overiooked. Our purpose was notto be exhaustive but iiiustrative of the state of the fieid. 
Tabie 1 summarizes how each statstacks up against our five principies. 
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Table 1 State Initiatives Compared to the Five Principals 



Florida 

Minnesota 

Washington 

Virginia 

Follow all 

matriculated 

students. 

Florida public universities 
only. Graduate cohorts of 
bachelor, masters, PhD 
recipients. 

Public community 
colleges, technical 
colleges, colleges and 
university. Exitcohorts of 
graduated students only. 

Undergraduate Cohorts of 
The Evergreen State 
College. 

Virginia public and private institutions 4 year 
and 2 year. Butnotall campuses 
participate. System tracks graduates only. 

Use standard 
data sources 
available In 
every state. 

Using "record linkage", 
student records and 
unemployment insurance 
wage files are matched 
anonymously 

Using student records and 
Unemployment Insurance 
wage file 

Using student records and 
Unemployment Insurance 
wage file 

Using student records and Unemployment 
Insurance wage file 

Create 

standard 

easily 

understood 

labor market 

experience 

measures. 

Annual record matching 
based on fourth quarter of 
the calendaryear 
(October-December). 
Earnings data is not 
adjusted for inflation. 

Reports hourly earnings 
for each of their 
educational programs 
reported. Three different 
points in time are: initial, 
six months, and two years. 
The "initial” point contains 
earnings data from the 
second quarter (October- 
December), six month 
contains data from the 
fourth quarter, and two 
year contains the tenth 
qua rterof earnings 
following graduation. 
Earnings are adjusted 
using the quarterly 
employment cost index. 

Earnings change overtime, 
1/3/5 years. Earnings 
collected forthe entire year, 
taking the highest reported 
employeronly. Forexample, 
if a match has two part time 
employers within a year, 
only the employer with the 
greatest reported wages is 
taken into account 

Earning calculated forthe quarters 3 to 6 
after graduation. Students earning less than 
minimum wage full time are excluded. 
Earning adjusted for inflation. 

Five yeartrends reported by field. 

Disaggregate 
data to the 
program 
level. 

Two types of 
disaggregation. 1) By 
degree type within each 
Institution, separated into 
percentage found by 
population 2) 

Disaggregated by program 
and degree type by 
institution. 

Program level information 
is available my institution 
only (i.e. Minnesota State 
University Moorhead - 
Accounting) 

Disaggregated by 
employment sector rather 
than degree program. BA 
vs. BS degrees (no MA, MS, 
PhD) 

Data can be displayed atthe campus or 
program level. No system level data. 

Make data 
publlcally 
available. 

Data available atthe 
Florida Department of 
Education website 
(www.fldoe.org/fetpip/). 
Otherdata available 
involving federal 
employment public 
assistance and 
corrections. Data available 
is starting with the 2007- 
2008 cohort 

Minnesota's earnings data 
iare available on the web 
via www.iseek.org, under 
"Plan Your Education". 
Then search fora specific 
institution, underthe 
programs offered sections 
then select "view program 
performance". 

Specific earnings data by 
program or major is not 
available but there are plans 
to make the data available in 
the future. Information about 
Washington’s methodology 
is available at 
(http://www.erdc.wa.gov/), 
under presentations. 

Data available through an interactive 
website. 

httD://esm.colleaemeasures.ora/esm/virainia/ 


Florida 

Students Followed 

Among the fifty states, Florida has one of the most extensive college graduate follow-up programs. Although the data are 
available for 2007 and forward, Florida has been a pioneer and has been working In the graduate earnings field for several 
decades. Florida's primary goal Is to find everyone who graduated from an educational system, K-20, matching not only earnings 
records, but also federal employment data, public assistance data and corrections data. At the higher education level Florida 
follows annual cohorts of graduates from Its public universities who attained a bachelor's, masters and/or PhD degree. 

Data Source 

Florida has coined their matching technique as "record linkage" which Is similar to that of other states. Using student records 
and unemployment Insurance wage files, they anonymously link graduated students to earnings records. Flowever, because of 
additional data provided by the state, they also gather Information from the Department of Corrections, Department of Children 
and Families, Department of Economic Opportunity, U.S. Department of Defense and U.S. Office of Personal Management. 
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Labor Market Experience Measures 

For its workforce deveiopment programs, Fiorida conducts "record iinkage" on a quarteriy basis, and for higher education it 
matches annuaiiy. The annuai record matching is based on the fourth quarter of the caiendar year (October-December). 
Earnings are not adjusted for infiation over time and therefore make year over year comparison difficuit. Earnings information 
avaiiabie on the web is for aii graduates starting with the 2007-2008 degree recipients. 

Data Disaggregation 

Once the earnings data are coiiected, Fiorida has disaggregated them in two different formats. The first format is broken down 
by degree type within each University, (i.e. aii bacheior degrees from the University of Fiorida). Those resuits are then separated 
into percentage found from various popuiation groups (federai empioyment, continuing education, pubiic assistance, etc). The 
second form of disaggregation is by various programs and degree types from different Universities. For exampie, bacheior's 
degree recipients within the accounting program from the University of Fiorida. 

Avaiiabiiity 

Fiorida has made its findings avaiiabie at the Fiorida Department of Education website (www.fidoe.org/fetpip/). Currentiy there 
are annuai cohorts starting with the 2007-2008 degree recipients. 

Minnesota 

Students Foiiowed 

Minnesota's graduate earnings system is geared towards potentiai students in their educationai system. Their system foiiows a 
vast majority of their graduated students and presents the data to coiiege seekers. Minnesota foiiows graduates from pubiic 
community coiieges, technicai coiieges, coiieges and universities as far back as 2006. 

Data Source 

Minnesota uses a method simiiarto both Fiorida and Washington, iinking student records to unempioyment insurance wage fiies 
to match students and earnings records anonymousiy. 

Labor Market Experience Measures 

Aiong with earnings data, Minnesota empioyers aiso report hours worked. Thanks to these additionai data, Minnesota reports 
houriy earnings for each of the educationai programs covered. Flouriy earnings are based off one quarter of reported data, either 
second, fourth or tenth depending on the reported point in time. These three different points in time are initiai six months, and 
two years from exit. The "initiai" point contains earnings data from the second quarter (October-December) foiiowing the end of 
the academic year. Six month retention contains data from the fourth quarter, and two year contains the tenth quarter of earnings 
foiiowing graduation. The quarteriy earnings reported are adjusted using the quarteriy empioyment cost index. 

Data Disaggregation 

Minnesota's avaiiabie information is directed primariiy towards potentiai students. Program ievei information is avaiiabie by 
institution oniy i.e. Minnesota State University Moorhead - Accounting. 

Avaiiabiiity 

Minnesota's earnings data are avaiiabie on the web via www.iseek.org, under "Pian Your Education". Then search for a specific 
institution; under the programs offered section; then seiect"view program performance". 

Washington 

Students Foiiowed 

Washington is in the preiiminary stage of their foiiow-up earnings program. They are conducting earnings matching on a smaii 
scaie in order to evaiuate and decide the future scope of their program. Currentiy they are using the bacheior's degree cohorts 
of The Evergreen State Coiiege. 

Data Source 

Simiiarto Fiorida and Minnesota, Washington uses student records and unempioyment insurance wage fiies to match students 
and earnings records anonymousiy. 

Labor Market Experience Measures 

Instead of quarteriy data, Washington uses reported yeariy data by the empioyer reporting the highest yeariy wage. For instance, 
if an earnings record contains two part time empioyers, the empioyer with the highest overaii reported wages wouid be used, and 
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the other wages are not added into the students' wage. Similar to Minnesota, Washington reports hours worked data as well as 
industry codes, which are reduced from six digits to two or three digits. Washington is not only interested in yearly earnings data, 
but also in evaluating earnings change through time at one, three and five year intervals. 

Data Disaggregation 

Currently Washington does not disaggregate data by program level butidoes disaggregate by degree type (BA vs. BS). 
Availability 

Specific earnings data by program or major are not available but there are plans to make the data available in the future. 
Information about Washington's methodology is available at the State of Washington Education Research and Data Center 
website (http://www.erdc.wa.gov/), under presentations. 

Virginia 

Students Followed 

Virginia's college earnings data are produced primarily by College Measures partnering with the American Institutes for 
Research. Together, they are following students graduating from Virginia's community colleges and universities. The system 
includes many public and private colleges but notall campuses in the state. Notably missing is the man campus of the University 
ofVirginia Non completers are notfollowed up ortaken into account. 

Data Source 

Similar to Florida and Minnesota, Virginia uses student records and unemployment insurance wage files to match students and 
earnings records anonymously. 

Labor Market Experience Measures 

Virginia's earnings data are captured for quarters 3-6 post graduation and are adjusted for inflation. Flowever students not 
earning more than the minimum wage of a full time employee are not included in the earnings data. That may pose a problem 
because some students working within the state may not show up. For example, graduates who only manage a part time job or 
graduates who are out of work may not be included within the earnings calculation, causing earnings to be greaterthen they may 
otherwise be. Five yeartrends in earnings are reported by field using CIP codes. 

Data Disaggregation 

Virginia's primary reason for measuring student earnings is to compare individual colleges and programs. Their data are 
disaggregated by university and by program on the College Measures website. 

Availability 

The College Measures website houses the earnings data for Virginia at http://esm.collegemeasures.org/esm/virginia/. 
Disaggregated earnings data can be viewed by selecting either a college or an area of study at the bottom of the page. 

D. THE NITTY-GRITTY OF TRANSPARENCY IN LABOR MARKET OUTCOMES 

In this section we delve into the details of creating the measures that meet the five principles outlined earlier. We recognize that 
once a standard measure of post-college earnings and employment exists, many different sorts of people will use it. Parents, 
students and the counselors that advise them will want to use these numbers to help make decisions about degrees, programs 
and colleges. University administrators, accrediting bodies, and state legislators will likely try to evaluate the effectiveness of 
education, in part, with these numbers. Many measures that are desirable for research purposes become undesirable when the 
general public misinterprets their meaning due to the subtleties involved in their calculation. In our proposed system, we place 
substantial value on providing a statistic that means what most users think it means. 

Problem: Unemploymentand missing earnings numbers 

When a student uses a post-graduation earnings number to evaluate a program or university, he or she will be interested in the 
whole labor market experience that follows education. Suppose, for example, that a business school claims that its 
baccalaureate graduates earn $60,000 five years after graduation. Most people would reasonably think this includes both the 
very successful graduates earning much higher sums and the much smaller earnings of people who found lower paying jobs or 
were unemployed. Many projects done with the base wage file only include students who had earnings reported in every quarter 
after graduation. Unfortunately, this amounts to assuming thata graduate is never unemployed. The average earnings number, 
then, is the average earnings among those graduates who have jobs for a full year. 
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Many reasons exist for having no reported earnings in the base wage fiie. The fiie derives its information from unempioyment 
insurance tax payments. Each quarter empioyers in covered industries must report what they paid empioyees to a state 
government agency (the Empioyment Deveiopment Department in Caiifornia). One route to having no reported earnings is to 
have a job that doesn't require empioyers to pay into the unempioyment insurance system. This inciudes peopie working in the 
miiitary, some Federai Government jobs, peopie in the underground economy^, as weii some seif-empioyed peopie. Another 
route to zero reported earnings invoives getting a job in another state. Of course the empioyer in the new state is iikeiy to report 
earnings to that state's unempioyment insurance tax coiiector, but states don't routineiy share this information with each other. 
Finaiiy, zero reported earnings might mean the person is unempioyed. 

Severai devices exist to minimize this probiem. We distinguish between two groups of peopie absent from the data set: 'Zero 
Earners" and "Missing Observations." and wish to inciude Zero Earners in our numbers, but not Missing Observations. A Zero 
Earner did not receive any money from empioyment. The government may ciassify the person as unempioyed (and provide 
some unempioyment insurance) or the person may receive no earnings (other than TANF or other government transfer 
payment). A "Missing Observation" is anyone the unempioyment insurance system in the State of Caiifornia knows nothing 
about. 


One can potentiaiiy recover a iarge number of Zero Earners by examining the unempioyment insurance ciaims fiie. For these 
individuais, one can be sure that they are unempioyed, and even report the amount they are receiving in unempioyment 
insurance and the number of weeks they have been unempioyed. Caiifornia's Empioyment Deveiopment Department is working 
to provide us with this information, but we have not yet received it. As a consequence, this information has not been 
incorporated into the numbers presented here. 

In our approach, we infer the presence of Zero Earners by making educated guesses based on individuais' successive quarters 
of earnings. Suppose we denote a quarter with positive earnings by an $ and a quarter with no reported earnings with a "M." 
Consider four patterns of quarteriy earnings. 


Figure 1: Earnings Records Inclusions and Exc 


iusion Criteria 



Ql 

Q2 

Q3 

Q4 

Q5 

Q6 

Q7 

Q8 

Q9 

QIO 

Qll 

Q12 

Decision 

Individual A 

$ 

$ 

$ 

$ 

M 

$ 

$ 

$ 

$ 

$ 

$ 

$ 

Included 

Individual B 

$ 

$ 

$ 

$ 

M 

M 

M 

M 

M 

M 

M 

M 

Excluded 

Individual C 

$ 

$ 

M 

M 

$ 

$ 

$ 

$ 

$ 

$ 

$ 

$ 

Included 

Individual D 

$ 

M 

M 

M 

M 

M 

M 

M 

M 

M 

M 

$ 

Excluded 


Individuai A has oniy one quarter of missing earnings information. In the quarters both before and after the missing number he 
or she is empioyed in a Ul-covered job in Caiifornia. It is uniikeiy (though not impossibie) that this person moved to a different 
state or took a job with an uncovered firm for a singie quarter. A fairiy good guess of their earnings (ignoring transfer payments) 
is zero doiiars. Individuai B is much more iikeiy to be a Missing Observation. In this case a good guess is the person moved out 
of state, or took a job not covered by unempioyment insurance. In the case of B, inferring that the person has 0 earning as 
opposed to simpiy being missing is uniikeiy to be correct. Individuais C and D iook a bit iike individuai A except that they have 
ionger periods of missing data. In our approach, individuals with short periods of missing data preceded and followed by 
reported earnings are put in the Zero Earner category, and included in calculations. We erred on the side of caution by defining 
"short" to be two consecutive quarters. If there were three or more consecutive quarters of missing earnings, the individual was 
eliminated from our data and treated as someone who could not be found using the Base Wage file. With this method we were 
able to find over 70% of the students we followed. 

Problem: Diverse paths through the educational system 

Following the labor market outcomes for all students means getting the Social Security number of every person attending the 
university. In addition to regularly enrolled students in baccalaureate and graduate programs, it includes people in professional 
and certificate programs (such as teaching credentials). For our purposes, "all students" really means "ALL STUDENTS." 
Tracking only a small number of students is a tried and true tactic for misleading others about the prospects of a college 
education. For example, colleges may tell a student that engineering graduates earn very high salaries after graduation. The 
number is true enough, but if our student has a weak background in mathematics, it may be very unlikely that he/she will earn an 
engineering degree, and may not earn a baccalaureate degree of any kind. 

Students take many paths through a college education. We've identified six paths that account for 100% of matriculating 
students: 

• Success Stories: students who started as first-time freshmen or transfers and graduated from the same campus. 
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• Other People's Children: students who started as first-time freshmen or transfers and graduated at a different institution, 

• Problem Children: students who started as first-time freshmen or transfers and never graduate from anywhere. 

• Stars: students who started as first-time freshmen or transfer students and completed a graduate degree at some other 
institution. 

• Graduate Completers: students who enrolled as graduate students and completed a degree. One twist here, students who 
enrolled as first-time freshmen and transfers at our campus and went on to complete a graduate degree at our campus are 
included in this group, and notin the undergraduate completers, as this represents their final exit point from campus. 

• Graduate Drops: students who enrolled as graduate students and left without completing a degree. 

The appropriate timing of labor market measurement differs for each of these groups. A "success story" enrolls in college, then 
graduates and enters the labor market immediately afterward. Hence, the moment the student leaves the institution, we can 
begin to follow his/her career. The group "other people's children" contains students that began college at the institution 
generating the list of Social Security Numbers, but transferred to another school and received a bachelor's degree there. They 
enter the labor market when they receive their degree, not when they leave their original institution. "Problem children" begin 
college, but never graduate. They "exit" the college system by giving up rather than graduating. 

Keeping track of these groups separately has important advantages. Importantly, members of some of these groups exit from a 
particular educational institution (California State University (CSUN) in our case) but don't enter the labor market right away. 
Thus, earnings numbers shouldn't be reported atX number of years after exiting CSUN. Instead, the numbers must be reported 
X years after exiting the educational system. Stars receive advanced degrees from other institutions. An earnings number one 
year after their exit from CSUN is likely to be what they make as somebody's research assistant at another university. This 
student is still in the educational system and has not yet entered the labor market. Fortunately, the National Student 
Clearinghouse provides data on all degrees received by students from member universitiesand covers 96% of all students in 
public and private US institutions. 

One added advantage of following all students is the ability to compare the earnings of "Success Stories"to "Problem Children." 
When you want to measure the value of college education in principle, you'd like to compare what a person going to college 
earned with what an otherwise identical high school graduate without college would earn. Of course, the "otherwise identical" 
part is a big problem. Comparing graduates to non-graduates gives some similar information, and is simple to compute. Since 
everyone in both categories was admitted to the university in question, they are meeting similar minimum entrance requirements 
and comparisons of earnings give an intuitive notion of the degree's value. 

Problem: Inflation 

In 1929 Babe Ruth was demanding to be paid $80,000. Even humble college professors make more than that today, but Mr. 
Ruth definitely had a better job. $80,000 in 1929 could buy goods and services roughly equivalent to a salary of $1,071,831 in 
2012. 15 Of course, most reported earnings will not be 83 years old, but correcting for inflation can still be critical. A person 
earning $80,000 in 2002 could buy as many goods and services as someone earning $101,880 in 2012. One could reasonably 
look at a report suggesting that graduates a decade ago earned $80,000 and grossly undervalue the education they received. 
The need for inflation corrections suggests that earnings numbers must be re-calculated regularly to identify all measures on a 
current dollar basis. 

Inflation corrections can solve other problems as well. Suppose you were interested in determining the average (or median) 
earnings of students who began studying at a university in 1995, five years after exiting the college system. Since they take 
different amounts of time to graduate, they enter the labor market at different times. With an inflation correction, we can get a 
sensible understanding of what happened to the entire group. 

Problem: Means, Medians and Cleaning the Data 

Statistical researchers have long realized that medians have some advantages over means as a transparent measure of 
earnings. A mean can be very high for a department because it produced a single very wealthy alumnus. Medians insulate users 
from this problem. We advocate the use of medians, but worry that using medians encourages the agencies preparing the data 
to avoid cleaning the data of fraudulent social security numbers and other problems. While we present means in this paper, both 
means and medians are planned foroursubsequentwork. 

While means and medians give a broad sense of typical earnings for graduates, they provide little information concerning the 
spread of the distribution. To better capture this we suggest providing the range of earnings between the 25th and 75th 
percentile rankings. While this approach was nottaken in the pilot project discussed in this paper, we do plan to calculate this in 
the follow up study currently in progress. 
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Fraud can be a serious problem with earnings data. In our pilot effort, one Social Security Number (SSN) showed up for as many 
as 50 employers in a quarter (never less than 17). Of 26 possible industries (as defined by 2 digit NAICS codesi^) this SSN 
showed up 13 or more times in each quarter. Quarterly earnings averaged $70,444”. It seems likely that considerably more 
than one person was using this SSN. We identified 20 SSNs with 14 or more distinct employers in a single quarter. Any 
statistical work done with these extreme values left in the data will generate seriously misleading conclusions. Medians, of 
course, minimize the impact of these problems. However, if we intend to make any other use of these data, the data need to be 
cleaned up. 

After investigating several methods of cleaning out bad SSNs, we decided to examine the shape of the frequency distribution. 
Figure one illustrates a stylized version of what we found. Note that only earnings in the top 1% of the data appear to contain 
significant amounts of anomalous earnings numbers. Hence our method of cleaning the data was brutally simple: we deleted the 
top 1% of earnings numbers. This allows for more accurate fitting of maximum likelihood models of earnings distributions based 
on common skewed distributions used with earnings (e.g., log-normal and similar distributions). Leaving the fraudulent data in 
would cause even greater problems. 

Of course, our approach may have eliminated a small number of very significant alumni of great interest to the development 
office as potential wealthy donors, but our data cannot be used to identify individuals so this is no great loss. If this procedure is 
adopted for use with the data provided to the California State University Chancellor's office, then we will recommend keeping 
them in a separate file for careful study of individuals.” 

Figure 2: Stylized Frequency Distribution for Quarterly Earnings Numbers 

frequency 



Problem: Some desirable jobs pay less 

People enroll in college in pursuit of various goals. While most of us are interested in future earnings as one goal, other goals 
can be very important. A program in social work is very successful if most of its graduates find work in that field. The fact that 
social workers earn less than MBA's is not a particularly damning criticism of the program. Ideally, we would like to have 
Standard Occupation Codes (SOC) to trackthe type of employment obtained. 

Unfortunately, the California unemployment insurance tax form does not collect this information. The form does collect North 
American Industry Classification System codes (NAICS). These can be useful in tracking employment in some industry-focused 
programs, but are not terribly valuable elsewhere. It is less useful for most programs on campus to know that an alumnus works 
for a firm that makes chocolate (the NAICS code information) than to know that his/her position is in senior management (the 
SOC code information). 

Since we can't track occupation codes, it seems most valuable to track the earnings potential of single programs over time. 
Such an approach allows cross-institution comparisons of alumni pursuing similar occupations. Earnings comparisons across 
disciplines with substantially different occupational goals are more problematic. 

Problem: Labor MarketCycles 

Alumni earnings numbers reflect more than the quality of the training received and the talents of the individual. They also 
represent labor market conditions. To be valuable, we track earnings over time so that the difference between the effect of 
training and the effect of job market conditions can be seen. Figure 2 presents a blatant example of this. Inflation adjusted 
earnings ofCSUN bachelor's degree recipients in computer science five years after graduation drop precipitously in 2001. This 
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reflects neatly the tech-bust occurring In that period. Almost certainly It has little to do with the quality of Instruction students 
received. Following these data overtime reveals that even the end of the tech-bust did not eliminate the reductions In earnings. 

Figure 3 CSUN Bachelors in Computer Science Graduates Earnings Year 5 Overtime (2010 Dollars) 



Problem: Rewards to Education Increase with time 

Preliminary estimates using our data suggest that earnings Increase with experience ata slightly decreasing rate. We estimated 
a fixed effects regression for the undergraduates using quarterly earnings as the dependent variable. More precisely: 

33153 

Earnings = 208.46 * Exp - . 25836 * ExpSquared + y~' 

i=l 

(-.2564) (.00898) 

Where Exp Is the number of quarters since labor market entry, ExpSquared Is the square of Exp and X; Is a dummy variable for 

student I and Is Its coefficiently numbers In parenthesis are standard errors for the coefficients. Economists have been 

running regressions of this sort since seminal work byj acob Mlncer^y, and our results are highly consistent It is worth noting in 
these Mincerian earnings regressions that the effect of education is typically smallest when initially entering the labor market In 
fact it is often true that that those going to college have lower earnings initially than those going to work right out of high school, 
since the high school graduate has a significant advantage in on-the-job training. The earnings profile for the educated worker is 
much steeper, however, and the more time that passes, the greater the earnings gap between the two. This suggests that 
focusing on relative earnings very close to entry into the labor market is likely to understate the reward to education. For this 
reason, we advocate ongoing matching of earnings to students over very long periods. 

OUTCOMES FOR CSU NORTHRIDGE 

We applied the five principles we introduced earlier to a project to follow-up students at our own campus, California State 
University Northridge (CSUN). Our campus is a large comprehensive public university in Los Angeles. CSUN's current 
enrollment is over 36,000 by head count and over 29,000 fulltime equivalent students. The student population is diverse 
ethnically and economically, and 11% of enrollment is atthe graduate level. 

Figures 4 through 6 present a brief summary of earnings outcomes for CSU Northridge students derived from our initial analysis 
of student earnings. The study tracked all students who entered CSUN between 1995 and 2000, a population of 44,437. The 
group includes first-time freshmen, transfer students, and post-baccalaureate students, but omits some students in certificate 
programs. Since the majority of exits from CSUN for this cohort occurred between 1997 and 2004 we have focused on this 
period. 

Figure 4 illustrates one of the most critical advantages of using unemployment insurance tax records to track earnings: you can 
find most people. We found earnings for between 60 to 80 percent of all students who exited during the 1997 to 2004 period. 
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Surveys of alumni earnings often generate response rates in the 10-15% range so this improvement is substantial. We do note 
that the percent found declines slightly overtime, typically 4% to 5% for each group. 

Figure 4 The Percent of Student Found with Earnings In California for Students Exiting Between 1997 and 2004 by Path 
Through Higher Education 
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Figure 5 presents earnings one and five years out for each of our 6 paths through the higher education system. Success stories 
earn about 30% more than problem children one year after entering the labor market, and about 32% more after five years. 
Hence, the reward for completing a degree is quite substantial. Other people's children earn 11% more than success stories after 
1 year and 9% more after 5 years. For those of us with graduate degrees, figure 3 also provides some reassuring evidence of 
the value of advanced degrees. Stars earn 52% more than success stories one year into the labor market and 28% more after 
five years. 
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Figure 5: Average Earnings for Students Exiting CSU Northridge Between 1997 and 2004 One and Five Years After 
Entering the Labor Market. (All numbers are in 2010 dollars) 



In Figure 6 we make anattemptto compare the CSUN numbers to national benchmarks, but the comparisons are crude at best. 
The Survey of Recent College Graduates is fairly comparable in that its reports all earnings - even when the number is zero due 
to unemployment, but it reports earnings only for the science and engineering fields it focuses on most. They also include many 
Ph.D. granting institutions whose training programs are quite different from those on a CSU campus. Nevertheless, Figure 6 
shows a comparison of the Surveys results relative to ours for Success Stories and Graduate Completers. Since the comparison 
is quite crude, we note only that our results for CSUN are fairly close to the national averages. 
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Figure 6: CSUN Earnings Compared to the Survey of Recent College Graduates during the same period. (All numbers 
are in 2010 Dollars) 



°iBase Wage AverageEarnings (Year 1 ) " Benchmark (Year 2) 


Table 2 illustrates our results for success stories across CSUN colleges differing by discipline. Comparisons across such 
colleges can be misleading, since they feed many different types of occupations, with different levels of pay. Doing these 
calculations every year would provide a significantly more useful comparison. For example, the Engineering College could 
compare graduates' earnings five years into the labor market now against a similar number five or ten years before, to assess 
the success of its programs. For most program level numbers, this is the most meaningful approach. 

Table 2 Success Stories Who Entered CSUN Northridge between 1995 and 2000 and Exited Between 1997 and 2005. (All 
numbers are in 2010 dollars) 



Avg Earnings 

# Found 

Totai # 

% Found 

Year After Graduation 

Yean 

Year 5 

Yean 

Year 5 

Year 1 

Year 5 

Yean 

Year 5 

Aii Groups 

$32,203 

$48,930 

12,114 

10,945 

15,900 

15,171 

76.2 

72.1 

Arts, Media and 
Communications 

$27,763 

$43,085 

1,957 

1,742 

2,618 

2,519 

74.8 

69.2 

Business and Economics 

$38,802 

$55,930 

3,088 

2,733 

4,031 

3,852 

76.6 

71.0 

Engineering and Computer 
Science 

$47,746 

$68,121 

505 

473 

696 

638 

72.6 

74.1 

Education 

$27,561 

$40,801 

144 

129 

178 

175 

80.9 

73.7 

Heaith and Human 
Deveiopment 

$29,140 

$47,328 

1,840 

1,677 

2,342 

2,245 

78.6 

74.7 

Humanities 

$28,771 

$44,248 

1,733 

1,626 

2,235 

2,119 

77.5 

76.7 

Sociai and Behavioral 
Sciences 

$30,002 

$46,004 

2,330 

2,111 

3,055 

2,935 

76.3 

71.9 

Science and Mathematics 

$28,022 

$47,833 

517 

454 

745 

688 

69.4 

66.0 


F. AN AGENDA FOR MOVING FORWARD 

The fact is California lags many states in creating a state-level data system for tracking students through the education system 
and linking those data to labor market information. It is particularly far behind in measuring the labor market outcomes of higher 
education. But with a new state law that allows the higher education segments access to the Unemployment Insurance Base 
Wage File, the state can create a strong system for measuring labor market outcomes. In this section we lay out an agenda for 
moving all segments of the higher education community forward together. 

Bring California's FI igher Education SegmentsTogetherata Policy Level 

A first step is to create a venue in which policymakers representing the three public segments (University of California, California 
State University and the California Community Colleges) and the private sector can come together to design a system. In the 
past, the California Postsecondary Education Commission (CPEC) would have been a natural body to do this, as it had 
representatives from all higher education segments. But it was essentially closed in the last round of budget cuts, so an 
alternative is needed. 
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We believe that a system that emerges through voluntary cooperation will be stronger in the long run than one that is imposed 
legislatively. We recommend that each sector designate a high level administrator to join an ad hoc group to develop a system 
that will ultimately be endorsed by the boards of each system. Private institutions should be represented as well and should 
encourage their colleagues to join in. Legislation may be required to compel for-profit colleges and universities to join the 
system. In the absence of CPEC, some other third party that is not part of any one segment, such as, an accrediting body (e.g. 
the Western Association of Schools and Colleges (WASC), or an independent research organization could convene the group. 

It seems to us that if the leaders of the public systems sought foundation funding for such an effort it would be forthcoming. 
These resources could be used to support a small professional staff to develop the technical aspects of the system. 

A Third Party to Calculate Basic Measures 

The power of third party evaluations is well recognized. If the proposed system is to have validity in the eyes of all stakeholders 
a third party, independent of any segment or campus, needs to operate it. Again, in the past CPEC would have been the natural 
group to build and maintain such as system. In its absence we recommend thatthe Labor Market Information Division (LMID) of 
EDD, which has been our partner in the pilot project, is a logical group to merge all the various data sets and produce the basic 
public measures of earning and employment. LMID currently generates the unemployment rate for the state, along with labor 
market demand forecasts and other key economic indicators so they have the needed expertise to manage the system. 

Develop Measures That Are Valid Across Segments 

Developing measures that can cut across the system is both a technical and political challenge. To have standard earnings 
measures that will work for Ph.D. graduates, undergraduate dropouts, and vocational certificate completers is not easy but can 
achieved, as we have demonstrated with our pilot study. Key decisions such as who is considered to be in the labor market, how 
earnings are adjusted for inflation and the appropriate follow-up period are key technical questions that must be addressed if 
data are to be comparable across segments, institutions and programs. Without agreement on these key measurement issues, 
we anticipate an arms race to come up with the measure that makes each segment or campus look the best. Achieving standard 
measures means bringing together researchers with the technical expertise and policymakers who able to commit their 
segments to common measures that are valid across the higher education spectrum. 

Access the National Clearinghouse and the Wage Records Interchange to Find People in Other States. 

As we note in our technical discussion, the power of our model is the ability of the system to find a large proportion of the 
students in question. This means not only finding out if students are continuing their education, but finding them even if they are 
employed out of state. To do this, the follow up system should use the National Clearinghouse to find students who have moved 
on to other higher education institutions in California and in other states. In addition, California's EDD Department should join 
the Wage Record Interchange System (WRIS). WRIS is a clearing house where states share their Base Wage File so thatthey 
can follow people into other states. If California participated, it could at least identify students who had left the state and are 
active in other states' labor markets. Currently 23 states are participating and the number is expected to grow. 22 Finally, other 
states, notably Florida, have gotten cooperation from the federal government in identifying students who are employed there. 
We believe California can get similar cooperation that will allow us to find even more students. In short, the proposed system 
should aim to identify as many students as possible to paint the most complete picture it can of students' labor market 
experience. 

Advanced Research Agenda Ideas. 

The measures proposed here, while valuable, only scratch the surface of what could be done with such a powerful data set. We 
recommend that after the basic measures are calculated and publicized, data sets be returned to system offices and campuses 
for research purposes. More in-depth analysis will yield many valuable insights into specific campuses and programs. Further, 
we suggest that sanitized data sets, which protect individual identities, be made available to researchers for more traditional 
research into the relationship between education and labor markets. 

Funding the System 

An initiative of the scope recommended here, while cost efficient in our view, is not cheap. Ideally, the state would fund the 
initiative, since it is a service not just to the institutions, but also to the entire state. California has missed out on two rounds of 
federal funding for large-scale education data systems, but it may be possible to build a labor market outcomes system into any 
future proposal, as previous federal proposals called for the inclusion such data. 

If producing these labor market outcomes becomes a requirement for the federal government, it would be cost effective for 
systems and individual colleges to pool their resources to have one centralized system rather than a host of campus and system 
initiatives. In this case, each campus could pay a per capita fee to support a larger system. 
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G. CONCLUSIONS 

Looking at the national policy landscape, it seems inevitable that eventually California higher education will have to become 
accountable for the labor market success of its students. We believe if higher education takes the initiative to develop a valid, 
reliable, cost efficient system that produces meaningful and understandable results, both the state's institutions and the public 
will be better off. We recognize that many of our colleagues may object to measures that so explicitly tie higher education to the 
labor market. So we will close by quoting Clark Kerr concerning the relationship betweenacademia and the economy. 

The cherished view of some academics that higher education started out on the Acropolis of scholarship and was 
desecrated by descent into the Agora of materialistic pursuits led by ungodly commercial interests and scheming public 
officials and venal academic leaders is just not true for the university systems that have developed at least since 1200 
A.D. If anything, higher education started in the Agora, the market place, at the bottom of the hill and ascended to the 
Acropolis on top of the hill... Mostly it has lived in tension, atone and the same time atthe bottom of the hill, atthe top of 
the hill, and on many paths in between . 23 
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